Controlling Text-to-Image Diffusion by Orthogonal Finetuning